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'nI" . Abstract 
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I We study an optimal control problem in Bolza form and we consider 

' the value function associated to this problem. We prove two veri- 
fication theorems which ensure that, if a function W satisfies some 
suitable weak continuity assumptions and a Hamilton-Jacobi-Bellman 

i -^H ' inequality outside a countably 7^^-rectifiable set, then it is lower or 

d • equal to the value function. These results can be used for optimal 

^ , synthesis approach. 
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1. Introduction. 

In this paper we consider a control system of the type: 

X = fit,x,u), uEU (1.1) eqintro 
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where x G is the state, f/ C is the control space and / is the controlled 
dynamic. Given a target 5* C M", a running cost L{t, x,u), a final cost iplt, x) 
and an initial condition {to,Xo), we consider the optimal control problem in 
Bolza form consisting in minimizing the integral of L summed with the value 
of ijj at final points for trajectories that start at Xq at time to and reach the 
target 5*. We define in the usual way the value function V(to, Xq) to be the 
infimum of the problem with initial condition (tg, Xq). It is well known that, 
under special condrtjgns, V satisfies the Hamilton- Jacobi-Bellman equation 
in viscosity sense and it is the unique solution. Part of the proof is based 
on the Dynamic Programming Principle. 

Therefore given a function W with suitable properties, it is possible to 
determine if W coincide with the value function, checking if it is a viscos- 
ity solution to the HJB equation. This type of theorems, called verification 
theorems, are useful, for example, when a candidate value function is pro- 



duced by means of the construction of a synthesis ||t8|| . It is then natural 
to ask for minimal conditions under which a function W coincides with the 
value function. If we know that W was obtained via a synthesis then the 
inequality W > V is granted by construction, thus we take this assumption. 
Then, for W to coincide with the value function, we prove it is sufficient that, 
outside a rectifiable set of codimension one, both W is differentiable and it 
satisfies a Hamilton- Jacobi-Bellman inequality in classical sense. Moreover, 
we make use of only some weak continuity assumptions, already usedin^f^ 
to prove ggt jjuality of a regular extremal synthesis, see Theorem p^" ^ij^d 
Theorem |6.i[ for details. A first result in this direction can be found in l|[Tl|, 
where the HJB inequality is asked outside a locally finite collection of regular 
manifolds of positive codimension (under more restrictive continuity assump- 
tions). Notice that, for an optimal control problem, if the value function is 
also semiconcave, it is differentiable outside a countably 7i"-rectifiable set. 



see 



We start considering the main assumptions for the problem and present- 
ing two technical lemmas, one of which dealing with the cardinality of the 
intersections between admissible trajectories and a countably 7i"-rectifiable 
set, while the other giving some conditions to assure the monotonicity of a 
real valued function. Also we state, without proofs, two propositions dealing 
with the properties of the solution to ( L*!] j and in particular dealing with 
existence, uniqueness^^d continuous dependence by data. 

Then, in Section ^ we recall briefiy the synthesis approach and various 
results available in the literature for comparison. Some examples of regular 
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optimal synthesis, to which our main results are applicable, are given. 

The first case we treat is the problem of finite time. We define a value 
function as the infimum, over all admissible trajectoriesre^ching the target 
in finite time. The main result of this part is Theorem p.i| which permits to 
verify if the function W is lower or equal than the value function. 

Nej^^we consider the infinite time problem. In this case the value func- 
tion (|6.1| ) is defined as the infimum of the cost functional over all admissible 
trajectories reaching^the target in infinite time. The main result of this sec- 
tion is Theorem |6. i| which gives sufficient conditions on the function W to 
ensure the inequality W ^ V, where V is the value function. In this case, 
for a technical reason, we consider a suitable neighborhood 5*1 of the target 
S and we suppose that the final cost ip is defined on S±M. order to give sense 
to the lirmt the definitifmof the value function ( p.il) . As a corollary of 
Theorem §^ and Theorem p!\\ we can treat a mixed case (see also | ^ ) , con- 
sidering at the same time the trajectories reaching the target both in finite 
time and in infinite time. iteni iiien2 

A key ingredient for Theorem pl\ and Theorem pl\ is the positiveness 
of the Lagrangian L, in order to prevent some bad phenomena such as the 
permanence of the system for an arbitrary interval of times in a region where 
X/ ij^n^ative making the value function equal to — oo as we see in Example 
bji. More precisely, it is not necessary to suppose L positive in the whol^ 
space, but some relaxed assumptions can be taken, as we see in Remark 04. 

This paper ends with an appendix^ where we give the defii^iyn of a 
non continuous viscosity solution as in and we state Theor er^ A. i ^ w. lyli 
ensures that, under suitable assumptions, the value functions (|274|) and (b7i|') 
are viscosity solutions to the Hamilton- Jacobi-Bellman equation. 
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2. Preliminaries. 

We consider a control system: 

x{t) = f{t,x{t),u{t)), {t,x)en, u{t)eU (2.1) 
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where 

(A-1) Q is an open and connected subset of M x M". 

(A-2) f/ is a non-empty subset of R'^, for some g ^ 1, g G N. 

(A-3) U = LP(]R; U) with 1 ^ j9 < +oo is the set of admissible controls. 

(A-4) f : Q X U — > M" is measurable in t, continuous in {x,u), differentiable 
in X and, for each u & U, Dj.f{-,-,u) is bounded on compact sets. 
Moreover there exists : M — > integrable and for every K, compact 
subset of Q, there exist a modulus of continuity ujk and a constant 
Lk > such that, if (t, x) E K and (t, y) G K, then for all u 

\f{t,x,u) - f{t,y,u)\ ^ ujkHx - y\) 
(fit, X, u) - f{t, y, u)) ■ (x-y) ^ LK\x-y 
\f{t,x,u)\ ^LK{^i{t) + \uf). 



(2.2) {f^ 



We consider a function L : f2 x f/ — > R and assume: 

(A-5) L is measurable in t and continuous in {x,u). Moreover, there exist 
ip2 ^ R^ integrable and, for every 0, Cr ^ such that 

\L{t,x,u)\^CRiip2it) + \uf), \{t,x)\^R. (2.3)^ 

In this paper we indicate with x{ - ;u,to,Xo) the solution to (^|) such that 
x{tQ;u,to,xo) = xq. Define the value function: 

T 

L 

to 



V{to,Xo):= inf <^ L{s,x{s;u,to,Xo),u{s))ds+ip{T,x{T;u,to,Xo)) 

{T,x{T;u,to,xo))eS 

(2.4) 

where S - the target - is a closed subset of R x R" contained in f2, : S" ^ R 
is the final cost. We recall the following definition: 



vf 



Definition 2.1 A subset A of is a countably Ti^ -lectifiahle set if there 
exist A\ and A2 such that A = AiU A2, Ai is a finite or countable union of 
connected submanifolds of positive codimension, andT-C"'{A2) = 0, where 
Ti^ is the k- dimensional Hausdorff measure. 
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3. Examples of syntheses. 

In next sections we give sufficient conditions for a candidate value function 
W to coincide with V. Beside some regularity conditions, we ask a HJB 
inequality outside a countably 7i"-rectifiable set. This regularity is shared 
by every function W obtained from a regular synthesis, thus it can be used 
to prove the optimality of the ^yn|;hesis itself. In this section we give various 
examples to which Theorem p.i| is applicable. First of all, we need some 
definitions. 

Definition 3.1 A synthesis T is a collection {{x(t,y){-),U(^t,y))}(^t,y)en such 
that x^t,y){-) = x{-; U(^t,y),i, y) : [t, r(t, y)] R", U{t,y) e U for every (i, y) G Vt, 
X{t,y){T{t,y)) e S and for every t e [t,T{t,y)] 

^{t,x<^m m (^) = '^m (s + t) a.e. 

and 

(*))(■) = + ^) 

Definition 3.2 A synthesis T is optimal if every U(^t,y) is an optimal control. 



There is a standare 
optimal synthesis, see 



^^od in geometric control theory to construct an 
"T?his consists of four steps: 1) using Pontryagin 



Maximum Principle and other geometric tools to study the properties of 
optimal trajectories, 2) derive a sufficient family of extremal trajectories (i.e. 
trajectories satisfying PMP), 3) construct a synthesis formed by extremal 
trajectories and 4) prove its optimality. In many cases, for autonomous 
systems, it happens that the extremal synthesis is associated to a feedback 
u : W" — > U that is smooth on each stratum of a stratification, see |^| 
for details. Roughly speaking a stratification is a locally finite collection 
of disjoint regular submanifolds, of various dimensions, that is a partition 
and such that the boundary of each manifold is union of manifolds of higher 
codimensions. In this cascj tl^e synt hesis is called regular in the sense of 
Boltyanskii-Brunovsky, see Ipj [?],' |lqf. 

Step 4) of the geometric control approach can thus be obtained in es- 
sentially two ways: either using the regularity of the synthesis, see or 



proving that the candidate value functiomPF associated to the synthesis co- 
incides with V. The latter is exploited in for a continuous W, defined on 
a subset of M", that is differentiable and satisfies the HJB equation outside 



6 



M. GARAVELLO 



a locally finite union of smooth submanifolds of positive codimension. Then 
the optimality is granted for initial points for which all admissible trajectories 

i ressan 

remains in the domain of W. A mild generalization is obtained in 1]^, where 
trajectories can exit the domain of W, but the boundary of the domain of W 
is a level set of W itself. Another approach is the one of nonsmooth analysis 



using which various verification theorem^ can ^be^ groved, see for example 



Our main results, see Theorems and |6.1| , generalize previous results 
in the following way: 

a essan 
we assume that W can be defined on a subset and the boundary 

of its domain is a level curve of W. 

2. We ask W to be differentiable and satisfy HJB only outside a countably 
7i"-rectifiable set. 

3. W is only lower semicontinuous (satisfying other weak continuity as- 
sumptions). 

A direct comparison with results of nonsmooth analysis is difficult. However, 
we point out that the value function fails in general to be locally Lipschitz 
continuous, see Example 3.1, for regular synthesis. In case of locally Lipschitz 
regularity, our result is corgem^ni^e^f^those obtained by nonsmooth analysis 
methods, see for example Ift |i9f.' 

We give now some examples to illus,trate the. applicability of our results. A 
whole class of examples can be find in Ip, [Lq] . The first example shows a typ- 
ical regular synthesis with a non locally Lipschitz continuous value function. 
In the second, the value function is not continuous and it is differentiable only 
outside a countably 7i"-rectifiable set. Last example shows the well known 
Fuller phenomenon. In this case optimal trajectories have an infinite number 
of switchings and the methods of Boltyanskii-Brunovsky do not work (while 
it does the result of f^). 

Example 3.1. Let a; G M and u G [—1, 1]. Consider the control system 

X + X = u 

and the problem of reaching the origin in minimum time. If we define Xi = x 
and X2 = X we obtain the following first-order system: 

X2 = —Xi + U. 



sinl 
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Every optimal trajectory is a bang-bang trajectory, i.e. formed by arcs (^orre- 
sponding to control +1 or — 1 . The synthesis is illustrated in Figure ere 
are some "switching curves": 



• all semi-circles of radius 1 contained in {(xi, X2) : 0:2 ^ 0} and centered 
at (2n + 1,0), with nEN\ {0}; 



• all semi-circles of radius 1 contained in {{xi,X2) : X2 ^ 0} and centered 
at {-2n - 1, 0), with neN\ {0}. 



Optimal trajectories switch along these curves, i.e. change control from +1 
to —1 or viceversa. Let 7^ be the trajectory that switches at points (±2,0) 
(defined say on [— oo,0]). Then the value function is not locally Lipschitz 
continuous at any point ^of supp (7^), but however it satisfies all the hy- 
potheses of Theorem Wfr M 



Example 3.2. Let = M^, f = 0, L = 1. Consider the target: 



S = {(t,x) ■.x^O,t = sin(l/a;)} U {x = 0, -1 ^ t ^ 1} U {t ^ 1} 
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Figure 2: Synthesis of Fuller phenomenon. 



Ful . Ph . 



and the final cost ip constantly equal to 0. The value function for this problem 
is given by: 



V{t,x) 



sin(l/x) 

1-t 



-1-t 




t if X 7^ 0,t ^ sin(l/x) 

if X 7^ 0, sin(l/x) < t < 1 

if t ^ 1 

if x = OA<-l 



if 



X 
X 



0,-1 < t < 1. 



This function satisfies all the hypotheses of Theorem and clearly it is not 
continuous. Moreover it is differentiable outside a countably ?^"-rectifiable 
set A, which is not a locally finite union of regular manifolds. < 

Example 3.3. (Fuller phenomenon). Let us consider the system 



Xi = X2 
X2 = U 



with \u\ ^ 1, Q 



S 



X {0}, iJ = 0, L{t,xujk2,u) 



X, 



This 



problem is well-known in the literature, see for example IQ. Every optimal 
trajectory is composed by an infinite number of bang-bang arcs, while the 
time for reaching the origin of is finite. There are two switching curves 
C"^ and C~ which separate into two regions and Z~ where the optimal 
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fin 



§ ul.Ph. 

Th^ value function of this problem satisfies all the hypotheses of Theorem 



4. Some useful results. 

We start by recalling without proofs some classical results about ODEs. 

Proposition 4.1 (Local existence and uniqueness of the trajectory). 

Assume (A-l)-(A-4)- Fixed u &U and (toja^o) ^ there exist 5 > and a 
uni (^^ hsolutely continuous function x{-\u, to, Xq) : [to,^o + ^] ~* IR" solution 

Proposition 4.2 (Continuous dependence by data). Assume (A-1)- 
(A-4)- Let (tQ,Xo) G Q, (tQ,Xn) G Q for every n G N and u E U, Un & U 
for every n G N. Let us suppose that there exists a time T > to such that 
x{-]u,to,Xo) and x{-]Un,to,Xn) are defined in [to, T]. //a;„ — > Xq and Un ~^ u 
in the strong topology of LP{[to,T];U) as n ^ +oo, then x{-;Un,to,Xn) —>■ 
x{-;u,tQ,Xo) uniformly in [to,T] as n ^ +oo. 

Now, we present two technical lemmas used to prove the theorems of the 
next sections. 

Lemma 4.1 Fix an element uj E U, t' < t" and x G M" with {t",x) G Q. 
Assume that there exists W, an open neighborhood of x in M", such that 
C^(-), the solution to (^{t) = f(tX^{t),uj) with Cy{t") = y, is defined on 
[t',t"] for any y eW and (t,C(t)) G Vt G [t',t"]. Let A be a countable 
Ti^ -rectifiable set. 

Then for a.e. y eW the set By := {t E [t',t"] : (t,C^(t)) G A} is finite or 
countable. 

This lemma is a slight generalization of a result proved in Theorem 2.14 



of Ifl^l, since here we consider the trajectory coupled with time. 

Proof. We can write A = A1UA2, where Ai = UjMj and {Mj}j^j is a finite 
or countable family of connected submanifolds of M"^^ of codimension dj > 0, 
and ?i"(A2) = 0. After replacing each Mj by a finite or countable family of 
open submanifolds of Mj, we may assume that the Mj are embedded. Define 
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lerv 



gel 



W ■.=]t',t"[xW and let $ be the map W 3 {t,y) ^ G ^. The 

Jacobian of $ is 

\ 



b 

v 



(4.1) Ijphi 



where b is the column vector f{t, (y{t),uj) and V''(t; t', Id) is the fundamental 
matrix solution to the linear system 



v{t) = -DJ{t,C{-t + t' + t") 



w] 



vit) 



(4.2) 



such that V^(t';t', Id) = Id. So the determinant of J$ is equal to the 
determinant of V''(t; t', Id), which is equal to exp j^, tr(—Dxf{s, C^{—s + t' + 
t"),cj))ds, by Liouville's theorem (see I|l3| J. In g|,rticular det{3^) is strictly 
positive for any t G [t',t"]. Moreover, by (A-||)" tr{—Dxf) is bounded on 
compact sets and then there exist c > 0, C > such that < c ^ det{3^) ^ 
C. 

So $ is a Lipschitz diffeomorphism. In particular we have 7Y"($^^(A2)) = 0. 
Now for each j consider Mj := $~^(Mj). It is an embedded submanifold of 
codimension dj > 0. Let 11 : W — W be the canonical projection. Consider 
the set Sj consisting of the points s G Mj such that Hi— is not regular. 



Mi 



0. 



Thus, by Sard's theorem, £"(0(5^)) = 0. Moreover W{U{<I>-\A2))) 
So the set B := n{^-^{A2)) U {\Jj U{Sj)) has Lebesgue measure in W. 
Let y eW\B. Then (t, (^{t)) ^ A2 ii t' < t < t" . To obtain the thesis, it is 
sufficient to show that, for each j, the set Ej = {t e]t',t"[: {t,(^{t)) e Mj} 
is at most countable. Fix j and suppose t G Ej. Mj has codimension 
dj > 0, so the dimension Uj of Mj is less or equal to n. Since y ^ B, the 
map dll{t,y) : T(^ y)Mj —>■ M" is onto, thus Uj = n and dll{t,y) is injective. 

Obviously dY\.{t^y){-^) = 0, so ^ ^ T^t^y^Mj and, consequently, (t, y) ^ if 
< |t — 1| ^ e for e > sufficiently small. Therefore t is an isolated point 
of Ej and so the lemma is proved. □ 

Lemma 4.2 Let g he a real-valued function on a compact interval [a, 6]. 
Assume that there exists a finite or countable subset E of [a, h] with the 
following properties: 

(a) liminf;,|o ^^""^^^'^"^ ^ for all x G [a, h[\E, 



lin 
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gc2 I (b) liminihio g{x + h) ^ g{x) for all x G [a, b[, 
( c) lim inihio g{x - h) ^ g{x) for all x e]a, h] . 
Then g{h) ^ g{a). 



For a proof of this lemma see Lemma B.l] 



5. Problem with finite time. 

se4 

We indicate with dQ the topological boundary of an arbitrary Q C M x R". 
Before stating the theorem we need the following definition 

Definition 5.1 Suppose that we have a time-varying Lip schitz- continuous 
vector field X on M" and : ^ M U {±00} . We say that W has the 
no downward jumps property (NDJ) along X if for any [a, 6] 9 t t— > 7(t), 
solution to 7(t) = 7(t)) such that {t,j{t)) E Q Wt E [a,b], we have 
liminihio W{t — h, ^{t — h)) ^ W{t, ^{t)), whenever t e]a, b]. 

teol Theorem 5.1 Suppose (A-^-(A-5) hold. Let Q C n be an open subset 
containing S. Let W : Q ^ W be a lower semicontinuous function such 
that: 

NUl i) W has the NDJ property along every time-varying vector field of the 
type f(t,x, u) with u E U fixed and for each t 

ess-liminfy^^W{t,y) ^ W{t,x). 

a) W on S. 
1 13 I Hi) At every point it,x) G dQ one has 

W(t,x) = sup W{s,y). 

14 iv) There exists a countably Ti^ -rectifiable set A Vt such that W is dif- 
ferentiable on Q\A and satisfies 

Ws{s,y)+ ini {Wy{s,y) ■ f{s,y,u) + L{s,y,uj)] ^ Q onQ\A. 
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v) L^O. 

Then W on Q. If Q = Vl we can drop hypotheses anc?^. 

Proof. Suppose by contradiction that there exists (toj^^o) G Q such that 
Wito^xo) > V{to,XQ). In particular V{to,xo) < +00. First of all, let us 
consider the case V{to, xq) > —00. So we can find e > 0, 6 > such that 



V{to,Xo) ^ W{to,Xo)-2e 
and, by the lower semicontinuity of W, 

\x - xo\ < 6 =^ W{to,x) > V{to,xo) + e. 



(5.1) 



(5.2) 



We can find u* E U such that x*{-) := x{-;u* ,tQ, Xq) satisfies (T, x*(T)) G S 
and 

e 



L{s,x*{s),u*{s))ds + iIj{T,x*{T)) ^ V{to,Xo) + 



(5.3) 



to 



Moreover, for every / G N there exists Ui eU such that — M*||LP([to,T]) ^ j, 
ui piecewise constant and left continuous. By 1^ Theorem IV. 9], there exists 
a subsequence of (m;)/, denoted again by {ui)i, and a function h G LP{[to,T]) 
such that \ui\ ^ h a.e. and ui converges to u* a.e. as / +00. Hence, if we 
denote by x/(-) the trajectory x{-;ui,T,x*{T)), for / sufficiently big, we have 
(see Proposition 4.2), 



and 



\xiit)-x*it)\<- VtG[to,T] 



[L{s,xi{s),ui{s)) - L{s,x*{s),u*{s))]ds 



to 



(5.4) 



(5.5) 



Fix / such that (p.4] ) and ( STSj) hold and an interval t"] such that Mi(t) = uj 
on Suppose that (t,x,(t)) G Q Vt G Let C^(t) be the trajectory 

associated to the constant control u such that C^{t") = y. By the fact that 
d{dQ, {{t,xi{t)) : t G [t',t"]}) > 0, we can find an open neighborhood W 
of xi{t") in such that (g^^y) G Q G W and {(t,C(t)) : t G C 
Q \/y E W. By Lemma \I.LI we have that for a.e. y E W the set i?^ := 
{t E [t',t"] : (t,C(t)) G A} is at most countable. 

Therefore, since for every fixed t ess-liminfy^^W {t,y) ^ W{t,x), then for 
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every Sj — > 0, Sj > there exists a sequence {y''j)j G N such that y^- — * xi{t"), 

W{t",yj) ^ W{t",xi{t")) + 6j and is at most countable. Consider the 
following function defined on [t',t"]: 



<^;.(t):=l^(t,C'(t))+ / Lis,CHs),^)ds. 

Jt' 



By the choice of i/j and the hypotheses ip''j is differentiable a.e. with 
a nonnegative derivative. By the lower semicontinuity of W anf 
condition, it follows that ip^j verifies the hypotheses of Lemma 
^'jit') ^ V]it")- Thus 



the NDJ 



71 and so 



W{t'X'^{t'))^W{t"X''^{t"))+ / L(.,C.(.),o;)cis 



(5.6) 



Now, using the fact that C^^{t") = y\ we obtain 



^ W{t",xi{t")) + 6,+ / L{s,C'{s),uj)ds. (5.7) 

it' 



7.1 



By Proposition 4.2, C^''(') 3;i(-) as j +oo and so by the Lebesgue 
theorem and the lower semicontinuity of W, passing to the limit as j +oo 
we obtain: 



W{t',xi{t')) ^W{t",xi{t")) + L{s,xi{s),uj)ds. (5.8) [8 



First consider the case {{t,xi{t)) : t G [toj^]} ^ Q- Summing ( p. 81) over each 
interval on which ui is constant we have 



Wito, xi{to)) ^ W{T, xi{T)) + / L(s, xi{s), ui{s))ds. (5.9) 

"'to 



9 
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Now, xi{T) = x*{T) by definition and so, using 



and 



W{to,xi{U)) ^ W{T,x*{T))+ L{s,xiis),uiis))ds 

J to 

^ ^{T,x*{T))+ [ Lis,xi{s),ui{s))ds 

J to 

£ 

^ V{to,xo) + -— / L{s, x*{s),u*{s))ds 



to 



T 



+ / L{s,xi{s),ui{s))ds 



to 



^ V{to,xo)+e<W{to,xi{to)). 

This is a contradiction. 

Suppose now {{t,xi(t)) : t G [^o,^]} 2 Q- Define 

f := inf {t ^ T : {s,Xi{s)) eQ Vs G [t,T]}. 



(5.10) 



tau 



In particular (f, a;/(f)) G dQ. Using the same argument to pass from (5^ 
to ( p. 91) , we obtain that for every r > f 



iy(r,a;Kr)) ^ iy(T,x*(T))+ / L{s,Xi{s)Ms))ds (5.11) 
and so, using ^ and (p.3|) 

iy(r,Xi(r)) ^ 7/;(T,x*(T))+ / L(s, xz(s), Mi(s))rfs 



bo 



^ V{to,Xo) + -- L{s,x*{s),u*{s))ds 



to 



Using (^), ( ^ ) and 



+ y L(s,Xi(s),Mi(s))(is. 
we obtain for all r > f 



(5.12) Ipos 



W{t, xi{t)) ^ V{to, xo)+e^ W{to, Xo) - e. 



(5.13) 



Ref 1 



Passing to the liminf as r ^ f and using the lower semicontinuity of W, we 
conclude 

W{f,xi{T)) ^W{to,xo) -e (5.14) [r^ 
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and so by ra) 



W{to,xo)^ sup W{t,x) !^W{to,xo) - e (5.15) |Ref3 

{t,x)eQ 



which is a contradiction. 

Now, we have to treat the case V{tQ,xo) = — oo. Since W{tQ,xo) > — oo 
and W is lower semicontinuous, we may find two constants M > 1 and 6 > 
such that: 

W{to,x) > -M 

for every x so that |x — Xo| < S. Moreover we can find u* & U such that 
:= M*, to, a^o) satisfies (T, x*(T)) G S and 

f L{s,x*{s),u*{s))ds + i){T,x*{T)) ^ -2M. 

With the same arguments of the first part of the proof we may find a con- 
trol ui E U piecewise constant and left continuous such that, if xi{-) is the 
trajectory x{-; Ui, T, x*(T)), 

\xiit)-x*{t)\<- yte[to,T] 



and 



[L{s,xi{s),ui{s)) - L{s,x*{s),u*{s))]ds 



to 



^ 1. 



Repeating the same calculations as before, we obtain that 
-M ^ W{T,x*iT))+ [ L{s,xi{s),ui{s))ds 

J to 

T 



^ ^Ij{T,x*{T))+ / L{s,x\s),u*{s))ds + l 
C -2M + 1 



to 



which gives M ^ 1, a contradiction. 

This concludes the proof of the theorem. □ 



Corollary 5.1 Let us suppose that W satisfies all the hypotheses of the pre- 
vious theorem. If moreover W ^ V then W = V. 
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Remark 5.1. If is produced by a synthesis procedure, the inequa! 
W ^ V always holds and so if W satisfies all the hypotheses of Theorem 
then W coincides with the value function. < 

Using the same techniques of the previous theorem, we can prove a corol- 
lary for value functions generated by approximated syntheses, and give a 
bound of the error thus produced. 

Corollary 5.2 Suppose (A1)-(A5) hold. Let Q (1 Q be an open subset con- 
taining S . Let W : Q ^ be a lower semicontinuous function verifying the 
ND J property along every time-varying vector field of the type f{t,x,u) with 
u & U fixed. Moreover we assume that, for each t, ess-liminfy^^W{t, y) ^ 
W{t^x) and that there exist e > and g G i^^(M), g ^ 0, such that: 

i) W ^%l) + e on S. 

a) At every point {t,x) G dQ one has 

W(t,x)= sup W{s,y). 

Hi) There exists a countably Ti^ -rectifiable set A C f2 such that W is dif- 
ferentiable on Q\A and satisfies 

Ws{s, y) + inf {Wy{s, y) ■ f{s, y, uj) + L(s, y, a;)} ^ -eg{s) onQ\A. 

iv) L ^ —eg. 
Then W + e{l + \\g\\i) on Q. 

Proof. Note that L{t,x,u) + eg{t) ^ and so 



W{to,Xo) ^ inf ( fl 



(T,x{T;u,to,xo))€S 

^ V{to,Xo)+e{l + \\g\\Li). 



T 

L{s, x{s; u, to, Xo),u{s))ds+ip(T, x(T; u, to, Xq)) 

to 



□ 
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Remark 5.2. Notice that the value function of an optimal control problem 
has the NDJ property along every possible direction as a consequence of the 
Dynamic Programming Principle. Indeed, for every {t,y) G Q \ S and for 
every admissible control u E U {in particular for every control u>Xi, where 
u E U and / bounded interval), the function 

/t+h 
L(s, x{s; M, t, y), u{s))ds + V{t + h, x{t + h; u, t, y)) 

is non decreasing for h G [0, 5] and S small enough. 
Instead, the hypothesis 

ess-limmiy^xW(t,y) < W(t,x) 

for each t fixed, says that, for every e > there exists a subset V {y E 
: \y — x\ ^ e} of strictly positive Lebesgue measure such that 

inf W{t,y) ^ W{t,x). 
yev 

So, if we consider a set Vi C of zero Lebesgue measure with cluster 
point, the set K\ Vi has a strictly positive Lebesgue measure. In the proof 
of Theorem p.l| this fact is used to avoid the points y for which is not 
countable. Moreover this hypothesis, coupled with the lower semicontinuity 
of W, gives the following: 

• for each t, 

W{t,x) = liminfW (t,y) = ess-limmiy^^W{t,y). 

y^x 



rip3 



[13 [teol 

Remark 5.3. Hypothesis ^ of Theorem p.i| says that, in the case Q ^ Q, 



the boundary of Q must be a level set of the function W. We can relax the 
same hypothesis in the following way: 



At every point {t, x) G dQ one has 



liminf l^(r, ^ sup W{s,y) 
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L(x,0) 








X 







Figure 3: L{x,0) of Example 5.1. 



and the conclusion of the theorem remains valid. Moreover if we define with 
R{t,x) the set of point reachable with an admissible control from {t,x), the 
previous condition can be replaced by 

inf W(s,y) ^W(t,x) 

{s,y)€R{t,x)ndQ 



and the conclusion still holds. 



The hypotheses of the positiveness of L is almost optimal as the next 
example shows. However, the Lagrangian L may be negative on some region 
if trajectories caft not stay for too longJii^jSuch a region and one can relax 
the assumption w as shown in Remark 



ExaLpos 



Example 5.1. Consider the system x = u, U = [—1, 1] and U = L^(M; U), 



n 



s 



R X {0}, Q = 1,1[ with the Lagrangian L{t,x,u) = 

v?' + x^ — Qx^ + (see Figure and ^/^ = on S'. Since the Lagrangian 
is negative in a region where the system can stay for an arbitrary interval of 
times, clearly the value function for this problem is equal to —oo. liW = C 



on Q wit 
Theorem 



iC negative cons 
dTO, but MT. In fact 



len W verifies all the. hypotheses of the 
Wt] are obvious, while m) holds because L 



is positive on Q and W is differentiable on Q. So there exist infinitely many 



VERIFICATION THEOREMS 



19 



functions W defined on Q verifying the hypotheses of Theorem 
which are not lower or equal to the value function V. 




Remark 5.4. If one wants to eliminate hypothesis ^ from the previous 
theorem, one may assume one of the following conditions: 

a) Fix e > and (t,x) G Q. We call ■ [i,T] — > M" an e-quasi optimal 
trajectory (e-q.o.t.) for {i,x) if: 

a.l 3u^eU such that Xs{s) = f{s,Xs{s),Ue{s)) for a.e. s G [i,T], 

a.2 Xsii) = X, 

a.3 (T,x,(T)) G S, 

a.4 V{i,x)+e^ L{s,X£{s),Us{s))ds + iIj{T,X£{T)). 

Now define Qi as the set of point (t, x) E Q such that, for every e > 0, 
there exists x^, an e-q.o.t. for (t, x), satisfying {s,Xir{s)) G Q for any 
s G [i,T]. What we need is that L ^ in f2 \ Qi- In fact, under 
this assumption, we may suppose that (s, G \ Qi for evg^^ 

s G [to, f[, where x is the trajectory defined in the proof of Theorem p.lj 
and the time f is defined in (^^). So the integral J^^ L{s, x{s),u{s))ds 
is positive. Otherwise we can assume Qi = Q. 



b) We can also use an hypothesis similar to one given in For any 

(t, x) G r2 and m G W, let Xt^xi''-, u) := x{-; m, t, x) be the solution to (271) 



associated to the control u. Consider the set P consisting of those 
points (t, x) of Q such that 

r 

/ L{s,Xt^x{s;u),u{s))ds ^ WT > t \/ueU. 

We have to suppose that, if (t, x) E Q \ [P U S], there exist a bounded 
and open set B, {i,x) eB<ZQ, BnS = ^, so that dB C Q, and a 
positive number M strictly less than 

inf {T >0 : d{{i+T, Xt,x{i + T; u)), dB) ^ d{{t, x), dB)/2} 
such that, for all m G W, (M + t, x^^iM + t;u)) e QnP and 
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L ^ on the whole space. 





1 







Example 5.2. Consider the system x = u, U = [—1,1], U = L^(K;U), 
= M+ X M, 5 = M+ X {0}, Q = M+x] - 1, 1[, = on 5 and the 
Lagrangian defined by 

{u^ + x'^ if X ^ 1 

(u2 + l)(2-x) + (x-l)(u2 + Ct) if l<x<2 
+ - 6x + 8 + Ct if x ^ 2 

It is clear that this Lagrangian, for C sufficiently big, satisfies the conditions 
a) and b) of the previous remark, even if it is not positive outside Q. ■< 

Remark 5.5. We can relax hypotheses ^ and ^ with the following: 
in') the boundary dQ is a level set of W; 
v') L ^ onn\Q. 

With these hypotheses, we can obtain an inequality of type ( p.8|) for each 
interval where the couple time-trajectory is in Q and then, using Hi'), v'), 
the lower semicontinuity of W and the NDJ property we can obtain (p.9|). <i 



6. Problem with infinite time. 

In this section we consider the control system ( 27L|) and assume that (A-^- 
(A-5) hold with < Cr < C for some C > and every i? > 0. Moreover 
we suppose that the target S" is a closed subset of R x R" which satisfies the 
structural property: 

(*) For any T > 0, there exists (t, x) G 5 with t ^ T. 

Let 5*1 be an open neighborhood of 5* contained in VL. Assume that the final 
cost ip is defined on 5*1 and, if d{{t, x{t;u,tQ, xq)) , S) ^ as t — >• +cxo, then 
the trajectory x{-]U,tQ,Xo) is definitively in Si, that is: 

(**) 3 T > t such that (s, x(s; u, to, Xq)) G 5*1 for all s ^ T. 
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Define the value function: 

f r+oc 

V{to,xo) := inf < L{s,x{s;u,to,xo),u{s))ds+limsupil>{t,x{t;u,to,xo)) 

d{{t,x{t;u,to,xo)),S)—i-0 
as t— >+oo 

(6.1) Ivfit 

In other words, we consider only the trajectories that approach the target S 
in infinite time. Notice that this condition does not imply that (T, x{T)) ^ S 
for any T ^ tg. 

Remcirk 6.1. The introduction of an open neighborhood of the target S 
is due to a technical reason and precisely to the fact that it is necessary to 
compare the candidate value function to the final cost near the target. Notice 
that in the following theorem the set Q must contain Si. For example we 
consider Q = R+ x R, = R+ x {0}, Q = {{t,x) : t > 0,x < 1/t} and 
Si = {{t,x) : t > 0, X < 3/t}. If {t, 2/t), with i > 0, is a trajectory, then it is 
definitely in 5*1, but it is never in Q. < 



teo2 I Theorem 6.1 Let Q C be an open subset containing Si. Let W : Q ^M. 
be a lower semicontinuous function such that 

i) W has the NDJ property along every time-varying vector field of the 
type f{t, X, u) with u & U fixed and for each t, 

ess-liminfy_^^W{t,y) ^ W{t,x). 

a) W on Si. 



ii2 



ii3 I Hi) At every point {t,x) E dQ one has 



ii4 



ii5 



W{t,x) = sup W{s,y). 
{s,y)eQ 

iv) There exists a countable HJ^ -rectifiable set A Vt such that W is dif- 
ferentiate in Q\A and satisfies 

Ws{s, y) + inf {Wy{s, y) ■ f{s, y, u) + L{s, y, u)} ^ mQ\ A. 

v) L^O. 
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Then W on Q. If Q = Q we can drop hypotheses and 

Proof. Suppose by contradiction that there exists (to, xq) G Q such that 
WitQ.Xo) > V(tQ,xo). In particular V{tQ,xo) < +00. First of all, let us 
consider the case V{tQ, Xq) > —00. As in the first part of the proof of Theorem 
p.i| , we can find e > and S > such that the following holds: 

V{to,Xo) ^W{to,Xo)-2e (6.2) 



\x-xo\<6 W{to,x) > V{to,Xo) + ^ . 



(6.3) 



We can choose u* G U, with the property that the trajectory {t,x*{t)) ap- 
proaches the target when t +00, and such that 

"+00 ^ 

L{s,x*{s),u*{s))ds + \imsnpilj{t,x*{t)) ^V{to,xo) + - , (6.4) 
to 2 

where x*{-) is the trajectory corresponding to the control u* such that x*{to) = 

Xq. 

Consider, now, a strictly increasing sequence of times Tj > to converging to 
+00. We may suppose that (t,x*(t)) G Q for every t ^ Ti. Fix j G N. 
For every / G N, there exists G U piec^ise constant and left continuous 



such that \\u^j — u*\\LP{[to,Tj]) ^ y- So, by Ifg, Theorem IV.9], we can extract 
a subsequence of (Wj)/, denoted again with (Mj)z, and we can find a func- 
tion hj G LP{[to,Tj]) such that ^ hj a.e. for every I G N and m^- u* 
for a.e. t G [to^^j] as / +00. Thus denoting with x^(-) the trajectory 



X ■; u 



^j,Tj,x*{Tj)), for / sufficiently big we have (see Proposition 3.2) 
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eq2 



|x^.(t)-x*(t)|<- 



Vt G [to, T,] 



(6.5) 



and then 



to 



[L{s,x^j{s) 



, u^j{s)) - L{s, x*{s), u*{s))] ds 



Now, fix / G N such that (|6.b|) and 
{{t,x''j{t)) : t G [to,^^]} ^ Q- So, usin^Lemmi 
arguments as in the proof of Theorem p.i| and 



hold. .First, let us .suppose that 




, Lemma [4.4 the same 
we conclude 



l^(to,4(to)) ^ W{T„x]{T,))+ L{s,x]{s),u]{s))d. 



to 



^ W{Tj,x\Tj))+ / L{s,x*{s),u*{s))ds + 



to 



liml 



(6.6) 
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Using ( jsT'dl ) and we have 

V{to,xo) + ^ < W{T„x*{T,)) + I ' L{s,x*{s),u*{s))ds + ^ . (6.7) 

Now consider the other case and precisely : t G [to,^^]} 2 Q- 

Define 

rj := inf {t ^ to : e Q Vs G [t,T,]} . (6.8) 

Given rj < t < T, 

iy(t,x;.(t)) ^iy(T,-,4(T,)) + y L(s,x5(s),4(s))rfs. (6.9) 

Considering the fact that {t,Xj{t)) {TpX^j{Tj)) as t ^ rj, {Tj,x^j{Tj)) G 
and (pii|) we obtain 

W{to,xo) ^W{T,,x*{Tj))+ L{s,x^j{s),u^^{s))ds. (6.10) 

We can now use the hypothesis ^7 (j^TSj ) and (^^) in order to have 

3e 

V{to,xo) + y < W^(^o,a;o) ^ 
^WiTj,x*{Tj))+ L{s,x*{s),u*{s))ds + - . (6.11)^ 

In all cases we have that, for every j G N, 

V{to,xo) + ^<W{T„x*{Tj))+ / L{s,x*{s),u*{s))ds+l . (6.12) 

So, applying the limsup as j — +oo we get 

V{to,xo) + — ^ limsup l¥(rj, x*(Tj)) + / L(s, x*(s), M*(s))rfs + - 

2 j^+oo Jtn 2 



'to 

~oo 



^ limsup iy(t, x*(t)) + / L{s,x*{s),u*{s))ds + ^ 
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For t sufficiently big, (t,x*{t)) E Si and so, using and 



3e 

V{to,xo)^ ^ \imsup^p{t,x*{t))+ I L{s, x'"{s),u*{s))ds + 



2 t— >+oo 

^ V{to,xo)+e 



to 



which imphes 



V{to,xo) ^ V{to,xo) 



which is a contradiction. 

It remains the case V^(to) ^o) = —oo. Since W{tQ, xq) > — oo and W is 
lower semicontinuous, we may find two constants M > 1 and 6 > such that 

W{to,x) > -M 

for every x so that |x — Xo| < S. Moreover we can find u* E U such that 
x*{-) := x{-]u* ^tQ^Xo) approaches the target when t — * +oo and 



to 



L{s,x*{s),u*{s))ds + \imsnpi){t,x*{t)) ^ -2M. 



Consider a strictly increasing sequence of times Tj > Iq converging to +oo 
and repeat the previous arguments in order to find a control m^- G U piecewise 
constant, left continuous and such that, if := x{-;u''j,Tj,x*{Tj)), 



|x;.(t)-x*(t)| ^- yte[to,T,] 



and 



[L{s,Xj{s),Uj{s)) - L{s,x*{s),u*{s))]ds 
Proceeding as before we obtain that 



to 



^ 1. 



-M ^ W{T,,x*{T,))+ / L{s,xUs),u){s))ds 



to 



^ W{T,,x*{Tj))+ / L{s,x*{s),u*{s))ds + 1 



to 
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for every j G N. Passing to the hmit we have: 

r+oo 

-M < \imsnpW{Tj,x*{Tj))+ / L{s,x*{s),u*{s))ds + 1 

r+oo 

^ \imsupW{t,x*{t)) + L{s,x*{s),u*{s))ds + 1 

t^+oo Jto 

r+oo 

^ hmsup^/'(t,x*(t)) + / L{s,x*{s),u*{s))ds + 1 

t~^+oo Jto 

^ -2M + 1 

which gives M ^ 1, a contradiction. 

So the theorem is proved. □ 



Corollary 6.1 Let W satisfies all the hypothecs of the previous theorem 
and moreover W ^ V where V is defined in (OJ. Then W coincides with 



the value function. 

Remark 6.2. In theorem p.l| the condition y) can be relaxed in the following 

way: 

lim sup W{t,x(t)) ^ \imsup4'{t,x(t)) 



t^+oo t^+oo 



for every x(-) solution to ( pTl|) such that d{{t,x{t)), 5") — >• as t — +oo. 



So, if one wants to minimize a Lagrangian cost without final cost, the 
condition becomes 

limsupVr(t,x(t)) ^ 

t-^+oo 

for every x{-) with the above property. < 

Remark 6.3. If we assume that there exists rj > such that S+B{0, rj) C 5*1, 
where 5(0, 77) is the ball in M""^^ centered in with radius 77, then hypoth- 
esis (**) obviously holds. In fact suppose d{{t,x{t;u,to,xo)),S) ^ as 
t +00. Then there exists T > such that d{{s,x{s;u,to,xo)),S) < ^ 
for all s ^ T. So we can choose an element {t{s),y{s)) G S" in order 
to have d{{s,x{s;u,to,Xo)),{t{s),y{s))) < ^ for all s ^ T. So the points 
(s, x{s; u, to, Xq)) E S + -8(0, rj) C S'l for every s ^ T. <\ 

2I JtfiD2 



Remark 6.4. We obtain a generalization of Theorems pTIl and ^^consid- 
ering the same problem ( pT]) with assumptions (A-^-(A-^, but we accept 
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at the same time all the trajectories that hit the target in finite time or t 
tencLto^the target in infinite time. Obviously an analogous theorem as 
and holds. 
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Ren ^ar ^ 6.5. Also in this case we can substitute hypothesis of Theo- 
rem p.il m aUp- analogous waj as in Remark 5.3. Moreover w^a^ eliminate 
hypothesis 0j of Theorem p. i| m the same way as in Remark 04. < 



A. Viscosity solutions and value functions. 

This appendix is intended to recall the notion of viscosity sub- and super- 
solution and to state sjomg icnown properties of the value function. Proof are 
analogous to those of 

Let fli be an open subset of M x M". We need the following definitions: 

Definition A.l Let f : A ^ W be a function where A is an open subset 
ofM}, for some / G N \ {0}. The lower semicontinuous envelope and the 
upper semicontinuous envelope f* of f are defined by: 

f^{x) := lim inf {f{y) : y E A,\y - x\ ^ r} , 
f*{x) := lim sup {f{y) : y e A,\y - x\ ^ r} . 

Proposition A.l The lower semicontinuous (resp. upper semicontinuous) 
envelope of a function f is a lower semicontinuous (resp. upper semicontin- 
uous) function. More precisely, it is the greatest (resp. least) lower semicon- 
tinuous (resp. upper semicontinuous) function less or equal (resp. greater or 
equal) to f . Moreover f is continuous if and only if f^^ = f*. 

Definition A. 2 We say that a lower semicontinuous function V : Qi ^ W 
is a viscosity super- solution to F{t,x, DtV, D^V) = in Qi if, for any ip G 
C^{Qi) and for any {to, Xq) G Qi point of local minimum for V — (f, one has 
F*{to, xq, Dtifito, xo), D^,Lp{to, xo)) ^ 0. 

Definition A. 3 We say that an upper semicontinuous function V : Qi ^ M 
is a viscosity sub-solution to F{t,x, DtV, D^V) = in Qi if, for any (p G 
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C^{Qi) and for any (to, xq) € Qi point of local maximum for V — ip, one has 
F^ito, xo, DfLpito, xo) , D^ip{to, xq)) ^ 0. 

Definition A. 4 We say that a function V : Qi is a viscosity solution 
to F{t,x, DtV, D^V) = in fli if is a viscosity super-solution and V* is 
a viscosity sub-solution to the equation. 

Remark A.l. Note that the notion of viscosity solution is not bilateral, in 
the sense that the set of viscosity solution to F = and —F = in general 
are different. < 

Let us consider the following hypotheses: 
(H-1) The functions / and L are continuous in all the variables. 
(H-2) f/ is a bounded set. 

We have the following: 



ppl Proposition A. 2 Let us asswme (A-l)-(A-5) and (H-l)-(H-2). Then the 
value function V defined in (^^) satisfies the dynamic programming princi- 
ple, that is 

V(to, xo) = inf If L{s, x{s; u, to, xo),u{s))ds + V(Ti, x(Ti; u, to, xo)) 



(T,x(T;u,to,xo))GS 



to 



teoA 



for every {to,Xo) & Q \ S and for every Ti less than the minimum time to 
reach the target. 

An analogous proposition holds for the value function V defined in ( jb^lp . 

Let us now state without proof the result that ensure that the value 
function is a viscosity solution to a Hamilton- Jacobi-Bellman equation. 

Theorem A.1 Let u^^a ^sume (A-l)-(A-5) and (H-l)-(H-2). Then the value 
functions (^.) and (OJ are viscosity solutions of 



-Vs{t, x) — inf {f{t, X, u) ■ Vy{t, x) + L(t, x,uj)} = in Q \ S. 
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